On Robust Arm-Acquiring Bandit Problems

In the classical multi-armed bandit problem, at each stage, the player has to choose one from N given projects (arms) to generate a reward depending on the arm played and its current state. The state process of each arm is modeled by a Markov chain and the transition probability is priorly known. The goal of the player is to maximize the expected total reward. One variant of the problem, the so-called arm-acquiring bandit, studies the case where at each stage new projects may arrive. Another recent extension of the classical bandit problem incorporates the uncertainty of the transition probabilities. This robust control problem considers an adversary, “nature”, who aims to minimize the player’s expected total reward by choosing a different transition probability measure each time after the player makes a decision. In this paper, we consider the robust arm-acquiring bandit problem, the combination of the two extensions above, and show that there exists an optimal state-by-state retirement policy. The extension to the robust arm-acquiring tax problem under some condition is also introduced.